(19) 



J 



(12) 



(43) Date of publication: 

29.04.1998 Bulletin1998/18 



FiirnnSiAchec; Patentamt | 
European Patent Office 
Office europ6en des brevets (11 ) E P 0 838 945 A2 

EUROPEAN PATENT APPLICATION 

(51) Int-a^: H04N5/44 



(21) Application number: 97116453.8 

(22) Date of filing: 22.09.1 997 



(84) Designated Ck)ntracting States: 

AT BECH DE DK ES R FRGBGR IE ITU LU MC 
NL PTSE 

(30) PriorHy: 25.10.1996 US 736982 

(71) Applicant: 

MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD. 
Kadoma-shI, Osaka 571 (JP) 

(72) Inventors: 

• Lopresti, Daniel P. 
Hopewell, New Jersey08525 (US) 



* Ma, YUe 

Plainsboro, New Jersey 08536 (US) 

* Tomklns, Andrew 

Pittsburgh, Pennsylvania 15218 (US) 

* Zfiou, Jian 

Plainsboro, New Jersey 08536 (US) 

(74) Representative: 

Steil, Christian, Dipl.-lng. et al 
Witte, Weller, Gahlert, 
Often & Steil, 
Patentanwdlte, 
Rotebuhlstrasse12l 
70178 Stuttgart (DE) 



CM 
< 
lO 

o 

CO 
CO 
CO 

o 

Q. 
LU 



(54) Video user's environment 

(57) The user { 1 02) communicates through a digitiz- 
ing writing surface (26) with the audio/video control 
apparatus (20). An on-screen display (32, 34) is gener- 
ated, providing the user (102) with a user environment 
in which a wide range of different tasks and functions 
can be performed. The digitizing writing surface (26) 
can be incorporated into a hand-held remote control 
unit (24) and the audio/video control apparatus (20) 
may likewise be incorporated into existing home enter- 
tainment or computer equipment. By tapping on the 
writing surface (26) a command bar (32) is presented on 
the screen, allowing the user (102) to select among var- 
ious functions. Included in these functions is an on- 
screen programming feature (158, 148), allowing the 
user to select programs for viewing or recording by entry 
of user<frawn annotations or commands via the writing 
surface (26). 
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Description 

Background and Summary of the Invention 

5 The present invention relates generally to the control of audio, video and multimedia equipment. More particularly, 
the irrvention relates to an on-screen user interface for interacting with audio, video and multimedia components using 
a remote control apparatus having a digitized writing surface for entry of hand-drawn instructions by the user. 

Television is on the verge of a revolution. Previously separate computer, communications and consumer electronics 
technologies are converging. This convergence will undoubtedly yield a rich assortment of program content and serv- 
10 ices» although it is by no means dear that a user will be able to navigate through the assortment of choices to find what 
he or she is interested in. For example, future systems are expected to provide both high quality digital, audio and video, 
up to 500 channels of programming, and a variety of on-demand services, including home shopping and banking, inter- 
active games and entertainment, multimedia libraries and full access to the Internet 

Providing a user interface for a complex system such as this is by no means a simple task. Easy-to-use access to 
15 a complex system - as television is expected to become - simply cannot be accomplished using the numeric keypad 
arxl fonrt^ard and reverse buttons on toda/s hand-heW remote controls. In terms of convenience and usability, present 
hand-held remote controls have already reached the point of diminishing returns. Adding more buttons makes these 
systems harder to control, not easier. Some systems today i^e on-screen display to echo the cunent operating param- 
eter of a remote control push kHitton as it is being pushed. While pressing the Color Tint button, for example, the con- 
20 ventional system may display a bar graph showing the current tint setting. While this sinple user feedback system is 
certainly better than nothing, it by no means solves the more fundamental problem of how to provide intuitive control to 
users of alt ages and all nationalities. Also, while the on-screen display of parameters may be viewat)le in a darkened 
room, the push buttons used to control these parameters may not be visible. Thus the greater the number of push but- 
tons on a hand-held remote, the harder it becomes to locate the correct push button while in a room darkened for opti- 
cs mai viewing. 

Aside from the shortcomings of push button user interface technology, cun^ent technology is also deficient in sup- 
porting users that do not have the time or inclination to learn complex system features or users, such as preschool chil- 
dren, who cannot read. The addition of a computer style keytx>ard for controlling the functions does not help to simplify 
such a system. Moreover, the placement of a keyboard on the family room coffee table appears less acceptable than a 

30 small remote control or digitized writing tablet. 

The present invention takes a fresh approach to the problem. Although the hand-held remote with push buttons 
may still be used, the present invention provides a digitizing writing surface through which the user may enter hand- 
drawn instructions. These instructions can be handwritten text, symbols or even pictures, all of which are written to the 
digitized writing surface using a pen or stylus. Such a means for controlling the system and providing input appeals to 

35 a broader range of users than does a conventional keyboard. Through the mechanism of provkling hand-drawn instruc- 
tions, complex systems can be controlled with ease. The user can create his or her own hand-drawn instructions 
(words, symbols, pictures, etc.) to represent any desired control function, even complex control functions such as 
instructing the audio/video system to turn on at a certain time and display the user's selected favorite program, or to 
search all available programs to locate those meeting the user's criteria of interest. This hand-drawn input can also 

40 include gestures which are recognized by the system and processed as commands to control various functions of the 
audioAndeo system. For example, drawing a large "X" over the digitized writing surface could be interpreted as a com- 
mand to turn off the television and/or the audio/video system. Additionally, handwritten symbols or text input can be writ- 
ten to the digitized writing surface and tiien processed using known handwriting recognition technology as if the 
symbols were typed on a keyboard. Once the handwriting is translated into standard character symbol codes, this input 

45 can be furtiier processed or stored in the system's memory for later use. 

According to one aspect of the Invention, the enhanced video user environment comprises an audio/video control 
apparatus that selectively performs predetermined audio/video control functions according to tiie user's selection or 
instruction. The control apparatus is preferably designed with a port fa coupling to a video display apparatus, such as 
a television, or projection system or monitor. The audio/video control apparatus can be packaged separately from the 

50 existing audio/video equipment, or it can be incorporated into existing components. A rennote control apparatus having 
a digitizing writing surface is provided for entry of hand-drawn instructions by the user. The remote control apparatus 
communicates witii the audio/video control apparatus. Alternatively, a full-featured personal digital assistant (PDA) that 
implements TV remote control as one of its programmable functions couW also be used as the remote control appara- 
tus. Many commerdally available PDAs cunently indude means for wireless communication, such as an infrared link. 

55 The system furtiier includes a processor that communicates with tiie audio/video control apparatus, the rennote 
control apparatus or both. The processor controls operation of the video display apparatus in accordance witii the hand- 
drawn instructions provided through the digitizing writing surface. The processor can be incorporated with the drcuitry 
of the audio/video control apparatus, or it can be incorporated with the drcuifry of the remote control apparatus. It is 
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also possible to inrplement the invention using multiple processors, one associated with the audio/video control, and 
another associated with the remote control. The multiple processors work in concert as distributed processors to imple- 
ment the processing functions required by the invention. 

For a rvoTe complete understanding of the invention, its objects and advantages, refer to the following specification 
5 and to the accompanying drawings. 

Brief Description of the Drawings 

Figure 1 illustrates a first embodiment off the invention in which the audio/video control apparatus is packaged as a 
10 set top box, suitable for use with a simple television set; 

Rgure 2 is another embodiment of the invention in which the audioAndeo control apparatus is packaged as part of 
a home entertainment system; 

75 Figure 3 is a close up perspective view of an exemplary renfK)te control unit with digitizing writing surface; 

Rgure 4 is a system block diagram showing the components of the invention together with examples of other com- 
ponents of audio/video equipment, illustrating how the invention is interconnected with this equipment; 

20 Figure 5 is a block diagram showing the hardware components of the audio/video control apparatus and remote 
control apparatus; 

Figure 6 is a block diagram of the presently preferred software architecture of the invention; 

25 Figure 7 is a diagram representing a screen snapshot, showing the command bar of the presently prefen'ed user 
interface; 

Figure 8 shows the sign-in panel of the presently prefen-ed user interface; 

30 Figure 9 shows an example of an ink search in the sign-in panel of the preferred user interface: 

Figure 10 illustrates standard television controls available for manipulation through the user interface by selecting 
the TV button on the command bar; 

35 Figure 1 1 illustrates an exanrple of a TV channel search using approximate ink matching; 

Figure 12 shows a TV program schedule as presented through the user interface; 

Rgure 13 shows a similar TV program schedule that has been limited to display onJy certain categories by manip- 
40 ulation through the user interface; 

Rgure 14 shows a VCR control function display produced by selecting the VCR button on the command bar; 

Rgure 15 shows an example of the video game quick access interface; 

45 

Figure 16 shows an example of the home shopping access interface; 

Rgure 1 7 shows an example of the ink mail (l-mail) user interface; 

50 Rgure 1 8 is a flow diagram describing the ink data interpretation that forms part of the recognition system; 

Rgure 19 is an entity relationship diagram illustrating the steps that tiie system performs in searching for a user- 
drawn entry or annotation; 

55 Figure 20 is a functional diagram illustrating the basic edit distance technique used by the preferred embodiment; 
and 

Figure 21 is another functional diagram illustrating how approximate matching may be performed with the edit dis* 
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tance technique. 
DesciiPtion of the Preferred Embodiment 

5 The present invention may be implemented as an audio/video system having an enhanced video user interface or 
video user environment. Many different inplementations are possible. Before proceeding vinth a detailed description of 
the system in its presently preferred form, an overview of two different implementations will be illustrated and described. 
These are sirrply examples of how one might implement the invention in a working system. Other systems are of course 
possible. 

70 Referring to Figure 1 the system of the invention is illustrated in a simple embodiment suitable for use with stan- 
dalone television sets or other less conplex home entertainment systems. As illustrated in Figure 1. the invention 
includes an audio/video control unit 20 that is packaged as a set-top box designed for placement atop a television 22. 
The hand-held remote control 24 includes a digitizing writing surface 26 on which the user may enter hand-drawn 
instructions using a suitable pen or stylus 28. A personal digital assistant (PDA) couW also be substituted or used in 

75 conjunction with remote control 24 and would include a digitizing writing surface and stylus. The control unit 20 and 
remote control 24 comnujnicate with one another via an infrared link depicted diagrammatically at 30. In this embodi- 
ment, the audio/video control unit includes a port on the rear of the unit (not shown) for coupling to the Video In port of 
television 22. In this way. the television 22 serves as a video display apparatus upon which the video user interface is 
prpjecled. In Figure 1 the video user interface has t>een shown in reduced detail as including a command bar 32 and a 

20 user interactive panel 34. The command bar 32 and panel 34 are projected onto the television screen (by inclusion of 
appropriate signals) with the existing NTSC video signals generated by the television tuner. Full details of the video user 
interface will be presented below. If desired, the control unit 20 may include a television tuner nrxxiule suitable for receiv- 
ing and decoding radio frequency television broadcasts via antenna or cable input. The tuner module supplies NTSC 
video signals to the Video In port of the television, bypassing the need to use the internal tuner section of the television. 

25 A more complex home entertainment system is shown in Figure 2. In this embodiment the renrrote control 24 is 
essentially the same as described in connection with Figure 1. The control unit 20 may be configured as a rack mount 
unit for inclusion in the home entertainment system, along with other conponents of audio/video equipment. For illus- 
tration purposes, the home entertainment system depicted here includes a large screen projection television 36, sur- 
round sound speakers 38. subwoofer 40 and multifunction tuner/amplifier 42. The tuner/anplif ier has video and audio 

30 irputs to which additional components of audio/video equipment may be connected. Illustrated here is a digital audio 
tape player 44. VCR 46. laser disc player 48 and camcorder 50. These are simply exarrples of tiie type of equipment 
that might be used with the present invention. Also included in the illustrated system is a personal computer 52. The 
personal computer may be connected to an Intemet service provider The control unit 20 is shown as a separate com- 
ponent in Figure 2 for illustration purposes. However, it is not necessary to package the control unit 20 as a separate 

35 conponent as illustrated here. Rather, the control unit may be incorporated into any of the audio/video components, 
including the television itself. 

An enlarged view of the remote control 24 is shown in Figure 3. The presently pref enred renx)te control 24 is housed 
in a hand-held case 54 having generally the same form factor and dimensions as a conventional hand-held remote con- 
trol unit. The remote control includes a conventional numeric keypad 56, VCR and laser disc motion control buttons 58 

40 as well as selected other buttons for providing convenient control of comrDonly used features. A thumb-operated jog 
shuttle wheel 60 may also be included for selecting various other system operating functions. Alternatively, a jog shuttle 
dial may be used in place of the thumb operated jog shuttle. 

The remote control 24 includes a digitizing writing surface 26 that Is designed to receive hand-drawn input through 
a pen or stylus 28. K desired, the digitizing wnriting surface can be hingedly attached to the case 54, allowing the writing 

45 surface to be flipped up to reveal additional push buttons beneatti. The digitizing writing surface 26 of the preferred 
embodiment is a passive screen that accepts pen stroke input (according to the ink data type described below) witiTOUt 
providing visual feedback on tine writing surface itself. According to this embodiment, the visual feedback appears on 
the video screen. One skilled in the art will also appredate ttiat digitizing writing surface 26 may be enribodied in a sep- 
arate tablet unit which can be placed upon a fixed surface, such as a tatrfe. allowing tiie tablet to be written to nnore com- 

50 fortably. Alternatively, the digitizing writing surface may be inplemented as an active screen that not only accepts pen 
stroke input but also includes a veritable display. The active screen may be bacWit so ttwt it may be viewed in the dark. 

An overview of the presently pretended system is shown in Rgure 4. Specifically, Rgure 4 illustrates the control unit 
20 and remote control 24 previously desaibed. The cortrd unit 20 includes a port 62 for coupling to a video display 
apparatus 64. As previously discussed, the display apparatus may be a television set or television nnonitor. or it may be 

55 a flat panel display, a projection system or a corrputer monitor. In most home entertainment systems tiie display func- 
tion is provided by tiie television. 

The audio/video control 20 may also be coupled to other equipment such as VCR 46. laser disc player 48 and mul- 
timedia computer 52. This is not intended to be an exhaustive list, as there is a wealth of entertainment and information 
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technology that can be coupled to the audio/video control 20. In Figure 4 this other equipment is shown diagramnnati- 
cally as other media 66. These media are preferably connected by conventional cabling 68 to the audio/video control 
20. Hie audio/video control thus operates as the audio/video signal switching and processing center for the system. For 
example, if the user has selected the VCR 46 as the source of program content, the audio and video signals from the 
5 VCR are switched through audio/video control 20 and communicated through port 62 to display 64. In this regard, the 
audio/video control 20 is preferably capable of handling multiple tasks concurrently. Thus the laser disc player 48 may 
be selected as the cunrent source of program material for presentation on display 64, while VCR 46 is taping a television 
broadcast for later viewing. The audioMdeo control nr^y include a television tuner to supply the necessary audio and 
video signals to the VCR. 

10 Whereas aixfio and video signal flow Is routed between components using cabling 68, the control functions can be 
provided via an atternate link such as an infrared link. In Figure 4 an infrared transponder 70 provides this function. The 
audio/video control 20 sends a command to transponder 70 and the transponder broadcasts that command to each of 
the components in the system. The infrared command includes a device header indicating which of the components 
should respond to the command. In one embodiment, the infrared link is bidirectional, allowing components such as the 

75 VCR 46 or multimedia computer 52, to send infrared replies back to the audio/video control 20. However, the infrared 
link may also be unidirectional, as with cunrent renrrote controls. There are, of course, other ways of communicating con- 
trol signals between the various components and the audio/video control 20. Infrared has the advantage of being com- 
patible with existing home entertainment equipment. By using infrared control, the audio/video control 20 is able to 
control the operation of home entertainment components that were designed before the advent of the present technol- 

20 ogy. Alternatively, the individual component may have infrared networking capabilities so that the remote control 24 can 
communicate directly with the components without having to go through the audio/video control 20. Thus tiie video user 
environment of the invention can be incorporated into existing systems, working with most of the user's existing equip- 
ment. 

The remote control 24 and control unit 20 preferably employ a form of distributed processing, in which each unit 

25 includes a processor that works in concert with tiie other. In Rgure 4 this distritxjted architecture is depicted diagrani- 
matically by processor 72, shown as being shared by or related to both tiie remote control 24 and tiie control unit 20. 
Alttiough distributed processing represents the preferred implementation, the video user environment could be imple- 
mented by a system in which all of tiie processing power is concentrated in one of the remote control or conti'ol unit 
devices alone. For exanrple, the remote control 24 could be constructed with minimal processing power and configured 

30 to sinpty relay all hand-drawn instructions of the user to the control unit 20 for interpretation. Such a configuration would 
require a higher data transfer rate between the remote control 24 and control unit 20. An alternate embodiment places 
processing power in the remote control 24. so that user-entered, hand-drawn instructions are interpreted in the renrote 
control unit, witii higher level instructional data being sent to the control unit 20 for further processing. 

Rgure 5 shows the hardware architecture of the preferred implementation. The components of the remote control 

35 unit 24 and the audio/video control unit 20 are shown in the dotted line boxes numbered 24 and 20, respectively. The 
remote control unit includes a processor 72a having local random access memory or RAM 74 as well as read only 
memory or ROM 76. While these functions are shown separately on the block diagram, processor 72a, RAM 74. ROM 
76 and various other functions could be implemented on a single, highly integrated circuit using present fabrication 
technology. Coupled to the processor 72a is an infrared interface 78. The remote control unit 24 may optionally include 

40 a push-button display 77 which provides visual feedback via various light functions and a push-button keypad 79 for pro- 
viding input to control unit 20. Push-button keypad 79 could have preprogrammed functions or may be programmed by 
the user, including a learning function which would allow keypad 79 to take on universal functions. Remote cont-ol 24 
may also be provided with a microphone interface 81 for receiving spoken commands from the user. One skilled in the 
art will appreciate that processor 72a or 72b may implement well-known voice processing technology for interpreting 

45 spoken commands into computer instructions. The remote control unit 24 also includes a digitizing writing surface com- 
prising tablet interface 80 and tablet 82. The tablet interface 80 decodes the user-entered, hand-drawn instructions, 
converting them into positional or spatial data (x.y data). Processor 72a includes an internal clock such that each x,y 
data value is associated with a time value, producing a record of the position of the pen or stylus as it is drawn across 
tablet 82. This space/time data represents the hand-drawn instaictions in terms of the Ink" data type. The ink data type 

50 is a defined data type having botii spatial and temporal components (x,y,t). The ink data type is described more fully 
below. 

The audio/video control unit 20 also includes a processor 72b having associated RAM 86 and ROM 88. Processor 
72b is also provkied with an infrared interface 90. Infrared interface 90 communicates unidirectionally or bidirectionally 
(depending on the embodiment) with infrared interface 78 of tiie remote control 24. In addition to tiie infrared interface, 
55 processor 72b also includes video interface circuitry 92 ttiat supplies the appropriate video signal to tiie video out port 
62. 

Much of the video user environment is preferably inrplemented as software that is executed by the distributed proc- 
essor architecture 72 (e.g. 72a and 72b). The architecture of this software is depicted in Figure 6. The software can be 
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stored in the read only menrories ROM 76 and ROM 88 of the remote control unit 24 and control unit 20. respectively. 
Alternatively, the software could also be downloaded to random access memories RAM 74 and RAM 86 over various 
transmission media, inclixiing but not limited to, standard telephone lines, fiber optic cable or the television cable that 
also delivers the video signals. 

5 Referring to Figure 6. the software conrponent of the invention is depicted diagrammatically at 100. As illustrated, 
the software component is situated t)€tween the user 102 and the hardware 104. The software provides each of the 
functions depicted generally at 106. 

The software component 100 has t>een illustrated here as the concatenation of several layers. At the lowest layer, 
closest to the hardware 104. is the hardware abstraction layer 108. This layer provides the connection to the actual 

10 hardware 104. The hardware atjstraction layer handles hardware-related issues such as implementing timers, tuning 
television tuners, supporting video and graphics adapter hardware, providing security functions and operating periph- 
erals. The hardware abstraction layer would, for example, include the necessary device driver for the tablet interface 80. 

One le^el atxwe the hardware abstraction layer is the microkernel layer 110. The microkernel layer serves as the 
real time operating system for the video user environment. The real time operating system employs drivers and iibrar- 

15 les, illustrated in layer 1 1 2. to produce the higher level input, video arxi network management functions. The user inter- 
face layer 1 1 4 is supported by the underlying layers 1 08. 1 1 0 and 112. Applications such as electronic program gukie. 
video player and multiuser games, are run within the user interface layer 1 14. An exemplary application is illustrated at 
116. 

20 Pretended Video User Interface 

The preferred video user interface, generated by user interface layer 114, is shown in Figures 7-14. 

Referring to Figure 7, the prefen-ed video user interlace presents a command bar 32 preferably at a predetermined 
location such as at the lower edge of the screen. The command bar provides access to various functions; the preferred 
25 command bar provides eight buttons for accessing those functions whose names appear on the buttons. Normally there 
is no indication that the video user environment is running on a particular video display device or television. During nor- 
nr^al viewing operation the video picture fills the entire screen and tiie command bar 32 is not present When tiie user 
wants to access the video user environment functionality, the user requests the command bar 32 by tapping the pen 
once anywhere on the digitizing tablet or pressing a button on the remote control unit 24 to make command bar 32 
30 appear on the screen. Another tap of the pen or press of the button causes the command bar to disappear. 

Anyone can walk up to a television equipped with the present invention and start using it immediately. However, 
much of the power of the video user environment comes from the ability to create personal annotations. For example, 
a user might draw a short descriptive pictogram to mark a favorite channel. 

Before such personalized data can be made available tiie user must identify himself or herself to the system. This 
35 is accomplished by selecting the "Sign In" button on the command bar by tapping it once. This brings up a panel shown 
in Figure 8 through which tine user may sign in. The panel comprises a user list 1 20 on which two types of information 
are displayed: a text string 122 and an associated ink region 124. The identity of each user is symbolized by tiie text 
string and its associate ink region. As illustrated, tiie ink region may not necessarily duplicate the text. In Figure 8 the 
text string JZ identifies the user who has signed her name as "Sophie" in the ink region. The ink region is entirely uncon- 
40 strained: it can be a picture, a doodle, a signature, a word written in any language and so fortii. There is explicit binding 
between tiie ink region and the text string, such that the bound pair is understood by both the system and the user as 
identifying a single individual. The linking of the ink region and the text st'ing forms a data structure often referred to as 
a tuple. This same paradigm carries through a number of the video user environment applications to be discussed. 

Once tiie Sign In panel is on screen the user may select an ID by tapping on it. Tapping the "Do It!" button com- 
45 pletes the action, logging in the user as tiie indicated ID. Alternately, the user may search for a specific ID using a 
searching feature of the invention discussed below. The searching feature uses an afi^^roxiniate ink matching tectviique, 
thus the user does not need to sign in precisely the same way each time. The system is flexible enough to accommo- 
date normal handwriting variations. 

The Sign In panel also offers tiie option of adding, deleting or editing a user ID. These operations are modal, mean- 
50 Ing that tiiey apply to a specific ID instance. Thus the "Edit" button is only active when an ID is selected. 

The system is capable of performing the approximate ink matching search on a user entered hand<lrawn annota- 
tion. By tapping on tiie Search button 126 a search dialog box 128 is presented as illustrated in Figure 9. The user 
enters a hand-drawn entry or anrK)tation in the ink region 130 and this entiy is compared with tiie ink data previously 
stored as user IDs. The approximate ink matching system of the invention identifies ttie best match and highlights it in 
55 the user list 120 as shown. If tiie user determines tiiat tiie highlighted entry is not correct, tiie user may proceed to the 
next best match by typing ttie "Find" button 132 again. The process can be repeated until the desired ID is found. 

As an alternate searching technk)ue, tiie user can search for the ID t>ased on the entry in the text string region 1 22. 
This is done by typing the desired text string using a soft keytx)ard brought up by tapping on the keytward icon 1 34. The 
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keyboard icon preferably appears as a standard QWERTY keyboard resembling a conventional Keyooard iuuiiu un a 
personal computer. When the keytxjard is used to enter a text string, the system finds an exact nnatch in the list of IDs 
by searching for the character string entered by the user Like the ink search, the text matching search can also be 
approximate. Thus if the user enters the query "ddP the text string "dpi" would be considered a better match than the 
5 text string "jeff." 

After the user has signed in with the user list saeen, a briefly displayed confirmatory screen is projected shewing 
the text and ink data representing the ID through which the user has signed in. Also, if desired, the time of day may also 
be momentarily displayed. After the confirmatory saeen has been displayed for a suitable length of time (e.g. five sec- 
onds) it disappears, leaving only the current video screen visible. In the event the user chooses not to sign in, the sys- 

10 tern assumes that the last entered user ID is applicable by default. 

The video user environment of the invention provides a full complement of standard television controls such as vol- 
ume, balance, tHrightness, color and so forth. In addition, an on-screen keypad is available for changing channels by 
direct entry of the numeric channel number or by "surfing" up and down the dial by clicking suitatrfe up and down txjt- 
tons. The standard television controls are presented by tapping the TV button 136 on command bar 32. 

15 The presently preferred implementation continues to use the traditional rerrwte control push buttons for performing 
standard television control functions such as those listed above. For continuity and maximum flextoility. these same 
functions are duplicated on saeen through the video user interface. 

Although the video user interface provides the same ability to control standard television control functions as the 
traditional remote control, the video user interface of the invention goes far beyond the traditional remote control. The 

20 invention provides sophisticated tools to help the user manage his or her video programming. Rgure 10 shows the tel- 
evision control panel 138 that is displayed when the TV control txrtton 136 is tapped. The numeric keypad 140 is used 
to enter television channels directiy and the up and down buttons 142 sequentially surf through the channels in fonward 
and backward directions. By tapping on the channel list button 144 brings up a scrollable list of channels with handwrit- 
ten annotations as illustrated in Figure 11 . As with the sign in panel, it is possible for the user to select an item manually 

25 or search for an item using the approximate ink or text matching techniques. In tiiis case, the numeric pad 140 
(accessed by tapping on tiie appropriate numeral icons) limits the user to numeric input (I.e. TV channels). Tapping on 
the "Schedule" button 146 displays a convenient television schedule illustrated in Figure 12. The prefen-ed implemen- 
tation portrays tiie TV schedule in the form of a traditional paper-based television guide. It has the distinct advantage, 
however, of knowing what time it is. Thus, ttie TV schedule screen (Figure 12) highlights programs currently playing, to 

30 assist tiie user in making a choice. Thus tiie TV schedule of Rgure 12 is an active schedule capable of highlighting 
which are current programs, updating the display in real time. In Figure 12 tiie active programs are designated by dotted 
lines at 148 to indicate highlighting. The present invention carries the concept of active scheduling one step further, 
however. Each program in the display is tagged with a predefined icon indicating its genre. Thus news, sports, drama, 
comedy, kids and miscellaneous may be designated. The user may limit the TV schedule to display only those pre- 
ss grams in certain genres by tapping the "Clear All" button 1 50 and by then activating one or more of the check boxes in 
the category pallet 152. In tiie example shown in Rgure 13, the user has elected to limit the display of programs in the 
sports, comedy and kids categories. This feature in the video user environment makes it much easier for the user to 
identify which programs he or she wants to watch. 

Finally, the TV schedule allows the user to program the TV to change channels at specific times automatically. Thus 

40 the user does not miss an important show. Unlike programming of current VCRs, which can be complicated and frus- 
trating, programming in the video user environment is handled in a highly intuitive way. The user simply taps on a show 
displayed in tiie schedule (such as "World Series" in Rgure 13), thereby highlighting it. Then, at the appropriate time, 
the video user environment switches to the proper channel (in this case channel 2). As witii all vkieo user environment 
applications, ease of use is key. 

45 The foregoing has descrtoed how the video user environment may be used to access and control television. Similar 
capability is provided for other audio and video components such as the VCR. Figure 1 4 depicts tiie VCR control panel 
154 that is displayed when the VCR button 156 is tapped. The VCR control panel provides traditional play, stop, pause, 
rewind and fast fbnward control. In addition, if the VCR equipnrtent is capable of such functionality, tiie VCR tape can be 
indexed fonvard or backward on a frame-by-frame basis. Similar capabilities can be provided for confrolling laser disc 

50 players, for example. 

As t>est illustrated in Figure 14, tapping tiie "Program" button 158 calls up a display visually identical to the TV 
schedule display of Figure 1 2. However, the TV schedule and the VCR schedule are maintained as separate data struc- 
tures, so that tiie user may program the TV and VCR independentiy. Using the same visual displays for different but 
comparable functions is one way tiie presently prefen-ed inrplementation makes the syst«n easier to use. By reusing 
55 the same icons and tools (including the same window layouts, locations and function of buttons) speeds the learning 
process, as the user only needs to have experience witii one instance of the tool to know how to apply it in its other 
settings. This also makes the video user environment application smaller, as code can be shared among several func- 
tions. 
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Tapping on the "Library" button 160 (Figure 14) brings up yet another browser displaying text and ink annotations 
in pairs. Similar in appearance to the channel list of Figure 1 1 . the video library displays entries that correspond to spe- 
cific video programs that the user can view at will. Thus the video library can serve as an interface to a video on deniand 
system or to recordings in the user's own personal collection. For example, the user might enter "Nightly News" in the 

5 video library keying it to a particular video on demand selection. Alternatively, the user may call up a memorable sport- 
ing event such as "Bob's Favorite Yankee Game." Thus the user could later search through the entries in the vdeo 
library and select an archived event by tapping on rt. This would in turn cause the video on demand system to com- 
mence delivery of the news or other entertainment program to the user. As video on demand systenns become more 
sophisticated, this capability can be quite valuable. For example, the user might wish to use the video library to review 

10 nightly news programs for the week he or she was on vacation and unat)le to watch the news. Or, the user might wish 
to use this video library to call up previous sporting events from the video on demand system. 

Tapping the "Games" button 162 (Figure 14) brings up a window (Rgure 15) that provides a quick and easy inter- 
face for a user (even a child) to access a variety of on-line games. Some of these games may involve other players on 
a network. The presentiy preferred emtxxliment of the video user environment does not directly implement any of these 

15 games, as it is contemplated that such games would be supplied by commercial software developers. The preferred 
interactive games interface simply displays a plurality of icons to represent each of the available games on the user's 
system. 

Tapping on the "Shopping" button 164 calls ip a display of home shopping options (Figure 16). Preferably each 
option is displayed as a separate icon that the user may tap on in order to access those shopping services. If desired, 

20 the shopping button could call a web site on the Internet tiiat could be used as a starting point for supplying hypertext 
links to other shopping locations. 

Tapping on the "l-Mail" button 1 66 (ink-mail) provides the user with an electronic rrtail communication system. In 
contrast with conventional E-mail systems that rely on keyboard-entered text, the video user environment allows the 
user to send hand-drawn or handwritten messages. The l-mail interface (Figure 1 7) preferably provides a notepad area 

25 into which the user can draw handwritten messages that may then be sent via tiie Internet or other suitable communi- 
cation network to a recipient. These handwritten messages allow for more personalized correspondence and are more 
accessible than typed electronic mail. Additionally, writing with a pen is more powerful. For exanrple, a user can begin 
writing an l-mail text message and tiien switch to drawing a map witfiout changing tools as is required with current key- 
board/nrouse-based electronic mail systems. 

30 As discussed above, the video user environment has access to a system dock whereby the TV schedule and VCR 
schedule are made active. The clock button 168 (Figure 14) may be tapped to call up a screen in which the user can 
set the correct date and time of day of the system. 

Prefenred Ink Search and Retrieval Technology 

35 

The preferred embodiment uses an approximate matching procedure to identify and rank possitMe hand-drawn 
"ink" entries made by tiie user using tiie digitizing tablet and pen. The approximate matching procedure is a fuzzy 
search procedure that identifies and ranks possible substring match candidates t>ased on a scoring and ranking dis- 
tance between the query and the candidate. The procedure produces a score for each candidate, allowing the candi- 
40 dates to be ranked in order of "goodness." 

One benefit of the approximate matching procedure is that any line breaks in the user-drawn entry or query have 
no impact on the ink search. Line breaks in writing are ignored, so that tiie user does not have to remember where the 
line breaks may have occurred in tiie original entry. 

The fuzzy search technique of the prefenred embodiment uses a vector quantized (VQ) representation of the user- 
45 drawn entry to capture and compare pen strokes of the ink data type. The ink data type is a system defined data type 
that captures the precise (X,Y) position of the pen tip over time as the user writes or draws an annotation or entry. Thus 
the ink data type captures not only the spatial position of the inK but also the temporal sequence over which tiie ink is 
"applied" as the user draws tiie entry on tiie digitizing writing surface. Figure 18 gives an overview of ttie manner in 
which pen stroke classification is performed using vector quantization. The ink data type records the motion of the pen 
50 tip over the surface of the digitizing tablet as a string of (X,Y) ink points. The individual (X,Y) ink points are sequentially 
captured, tiiereby preserving the tenporal or time-based component of the data. Thus the ink data type may be consid- 
ered as comprising (X.Y.T) vectors. 

As illustrated in Figure 18. the incoming ink data 200 are broken into strokes as at 202. Segmenting the ink data 
into strokes allows each stroke to be analyzed separately By way of illustration, Figure 18 shows that the plus sign {+) 
55 in the incoming data 200 was drawn by the user, first fonning a horizontal line and tiien forming a vertical line. This is 
illustrated at 202 by reading the segmented data at 202 from left to right. 

After stroke segmentation the individual strokes are then analyzed to extract feature vectors. This is shown dia- 
grammatically at 204. In Rgure 18, ttie extracted feature vectors are shown graphically to simplify tiie presentation. In 
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the actual embodiment, the extracted feature vectors are represented as numerical data that is stored in the computer. 
As indicated at 206, each extracted feature vector is classified according to a predeternnined code book 210. The pres- 
ently preferred embodiment stores 64 clusters of stroke types, each cluster being represented by its centroid a average 
stroke of that type. As in the case of the extracted feature vectors (block 204) the feature vector clusters are stored as 

5 numerical computer data, in Rgure 18 the data comprising code book 210 are shewn graphically (instead of numeri- 
cally) to simplify the presentation. In Rgure 18 note that the horizontal line segment of block 206 most closely matches 
the centroid 212 of the Type 2 stroke cluster 214. Thus in the output string (block 216) the VQ code 2 is used to repre- 
sent the horizontal line in block 206. In block 216 the leftmost numeral 2 corresponcte to the leftmost horizontal line 
stroke. The remaining codes represent the renrwining ink strokes comprising the original incoming ink data. 

10 Through the above-described procedure the incoming ink data is converted, pen stroke by pen stroke. Into a feature 
vector that conresponds to each individual pen stroke. The set of feature vectors which collectively represent a series of 
pen strokes are stored in the computer database as the user-drawn annotation. This is depicted at 218. 

To further illustrate, a software block diagram of the presently preferred embodiment is shown in Rgure 19. The 
annotation system operates on digitized pen stroke data that is ultimately represented as an Ink^ data type. As will be 

15 illustrated, it is not necessary to convert the ink data type into an ASCII character data type in order to perform the 
search and retrieval procedures. Indeed, in the case of graphical (nontext) annotations, conversion to ASCII would have 
no meaning. Thus, a significant advantage is that the annotation system operates in a manner which allows the "ink" 
data to be language-independent. 

Illustrated in Rgure 19, the user-drawn query 300 is captured as a string of (X.Y) ink points, corresponding to the 

20 motion of the pen tip over the surface of the digitizing tablet or pad as the user draws query 300. The presently preferred 
embodiment digitizes this information by sampling the output of the digitizing pad at a predetermined sampling rate. 
Although a f ixed sampling rate is presently prefen-ed. the invention can be implemented using a variable sampling rate, 
as well. By virtue of the digitized capture of the X,Y position data, both spatial and temporal conponents of the user- 
drawn pen strokes are captured. The temporal component may be inplicit information - the ordering of sampled points 

25 relative to one another conveys temporal information. Altematively the temporal component way be explidt - the exact 
time each point was sampled is captured from an external dock. 

In the presently preferred embodiment, employing a fixed sanrtpling rate, each X,Y data point is associated with a 
different sampling time. Because the sampling rate is fixed, it is not necessary to store the sampling time in order to 
store the temporal data associated witii the pen stroke. Simply recording the X,Y position data as a sequence automat- 

30 ically stores the temporal data, as each point in the sequence is known to occur at the next succeeding sanrpling time. 
In the alternative, if a variable sampling rate system is implemented, (X.YT) data is captured and stored. These 
data are the (X,Y) ink points and the corresponding time T at which each ink point is captured. 

The raw ink point data is stored in data store 302. Next, a segmentation process 304 is performed on the stored ink 
point data 302. The presentiy pref enred segmentation process searches the ink point data 302 for Y-minima. That is, the 

35 segmentation process 304 detects those local points at which the Y value coordinate is at a local minimum. In hand- 
drawing the letter "V" as a single continuous stroke, the lowermost point of the letter "V would represent a Y-minima 
value. 

Segmentation is performed to break the raw ink point data into more manageable subsets. Segmentation is also 
important lor minimizing the variation in ttie way the users produce ligatures; the connection of characters or even 
40 words. These segment subsets may be designated using suitable pointers to indicate the menx)ry locations at which 
the Y-minima occur. In this case, tiiese segmentation pointers may be stored at 306 to be assodated witii the ink point 
data 302 previously captured. In tiie alternative, if desired, ttie segmented data may be separately stored in one or nnore 
memory buffers instead of using pointers. 

Once the raw data has been segmented the individual segments or pen strokes are operated on by a set of extrac- 
ts tion functions 308. The presently preferred embodiment operates on the pen stroke (segment) data using 13 different 
extraction functions. These extraction functions each extract a different feature of tine pen stroke data that are then used 
to construct a feature vector. Table I lists the presentiy preferred features ttiat are extracted by the exh-action functions 
308. For further background information on these extraction functions, see Rubine, Dean, ''Specifying Gestures by 
Example." Computer Graphics, Vol. 25, No. 4. July 1991. The feature vectors of a given stroke are diagrammatically 
50 represented in Rgure 19 at 310. 
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Where P represents the total number of points. 

The extracted feature vectors represented at 31 0 are then coded or quantized by comparison with a predetermined 
set of dusters of stroke data types. The feature vector data 310 is quantized by vector quantization process 312 to 
assign each duster to the closest predetermined stroke type. In this regard, the presently preferred embodiment 
50 defines 64 different stroke types that are each represented by a different name or number. Afthough the presently pre- 
ferred system uses 64 different stroke types, the principles of the invention can be errpk>yed with a greater or fewer 
number of stroke types. 

The predetermined stroke types are arrived at during a training procedure 313. The training procedure may be 
used to predetermine a vector quantization (VQ) code book 314 that is then used for multiple users. In many commer- 
55 cial implementations it will be desirable to train the system at the fadory. using a set of user-independent training data. 
Alternatively, the training procedure can be used prior to use by an individual user. Both applications work well. In either 
case, the system is still user-dependent because there can be a great deal of variation in the way two different people 
draw the same annotation. Thus the preferred emtxxiiment is best suited to searching one's own annotations. 
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It will be appreciated that in vnosX cases the user will not draw the same annotation in precisely the same way each 
and every time. That is, the (X.Y.T) coordinates and temporal properties of a given annotation nray vary somewhat, 
each time the user draws tiiat annotation. The presently preferred system accomnrxxiates this variation first by the man- 
ner in which the vector quantization is performed. Specifically, the vector quantization process 312 assigns each input 

5 stroke to the predetermined vector 315 from the user-dependent stroke types 314 tiiat represents the closest match. 

After each of the strokes representing the query has been processed in this fashion, a comparison is made 
between those strokes and the user-drawn annotations that have been stored in association witii the documents in the 
database 320. Thus, for example, the query "important" may be conpared against the stored annotation "this is very 
important!" An edit distance analysis is performed to make this comparison. 

10 Shown as edit distance analysis process 318. the query stroke type string is compared with each of the stored 
annotation stroke type strings 321 of the database 320. The edit distance analysis compares each stroke type value in 
the query siring vwth each stroke type value in each of tiie annotation strings. A edit distance computation is perfomned 
by this comparison, yielding the "cost" of transforming (or editing) one string into the other. The individual string/string 
comparisons are then ranked according to cost, with the least cost resultants presented first. In tiiis way, a sorted list 

15 comprising all or the n-best matches is displayed in the thumbnail sketches of the main browser screen. Alternatively, 
rather than showing a sorted list, the user may be shown the best match on the main browser screen. If the user deter- 
mines that tills match is not connect, the user may tap the "Next" txrtton (not shown) to see the next best match. 

Figure 20 shows the basic edit distance technique. In tiiis case, the stored annotation "compress" is compared with 
the query sti^ing "corrpass." It should be understood that Rgure 20 depicts the conparison of two strings as a conrpar- 

20 ison of iridividual letters in two differently spelled words. This depiction is intended primarily to aid in understanding the 
edit distance computation technique and not necessarily as a depiction of what two stroke type strings might actually 
look like. In this regard, each of the 64 different stroke types may be ariDttrarily assigned different numerical labels. Thus 
the edit distance confutation would compare the respective numeric labels of the stored annotation and the input 
query directly with each other. There is no need to convert the individual strings into ASCII characters and Figure 20 is 

25 not intended to imply that such conversion is necessary. 

Referring to Figure 20, each time the annotation string stroke value matches the query string stroke value a cost of 
zero is assigned. Thus in Figure 20, a zero cost is entered for the comparison of the first four string values "comp." To 
accommodate the possibility that a string/string comparison may involve insertion, deletion or substitution of values, a 
cost is assigned each time an insertion, deletion or substitution must be made during the comparison sequence. In tiie 

30 example of Figure 20, the query string "compass" requires insertion of an additional value "r" after tiie value "p." A cost 
of one is assigned (as indicated at the entry designated 422). Continuing with the comparison, a substitution occurs 
between the value "e" of tine stored annotation string and the value "a" of the query string. This results in an additional 
cost assignment of one being added to the previous cost assignment, resulting in a total cost of two, represented in Rg- 
ure 20 at 424. Aside from these insertion and substitution operations, tiie remainder of the comparisons match, value 

35 for value. Thus, the final "cost" in comparing the annotation string with the query string is two, represented in Figure 20 
at 426. 

In the preceding discussion, a first minimum cost path was described in which "compass" is edited into "compress" 
by inserting an "r" and substituting an "e" for an "a." An alternative edit would be to substitute an "r" for an "a" and insert- 
ing an "e." Both of these paths have the same cost, namely two. 

40 Rgure 21 gives anotiier example of the edit distance computation technique. As before, strings of alphabetic char- 
acters are conpared for demonstration purposes. As previously noted, this is done for convenience, to simplify the illus- 
tration, and should not be interpreted as implying that the strings must be first converted to alphanumeric text before 
the comparisons are made. Rather, the procedure illustrated in Rgures 20 and 21 are performed on the respective 
stroke data (vector quantized symbols) of the respective stored annotation and input query strings. 

45 Figure 21 specifically illustrates the technique that may be used to perform an approximate match (word spotting). 
In Figure 21 the stored annotatbn "This is compression," is compared with the query string "compass." Note how the 
matched region 430 is extracted from the full string of the stored annotation by scanning the last row of tiie table to find 
the indices tiiat represent the lowest value. Note tiiat tiie first (initializing) row in Figure 21 is all Os • tiiis allows the 
approximate matching procedure to start anywhere along the database string. 

50 The presently preferred edit distance procedure is enhanced over the conventional procedures descrit>ed in the lit- 
erature. In addition to the tiiree k)asic editing operations (delete a character, insert a character, and substitute one char- 
acter for another), *rt is useful to add two new operations when comparing pen stroke sequences. These new operations 
are "spirt" (substitute two strokes for one stroke) and "merge" (substrtute one stroke for two sfrokes). These additional 
operations allow for errors made in stroke segmentation and generally leads to more accurate results. 

55 The use of our enhanced edit distance procedure is illustrated in Rgure 21 . In Rgure 21 the spirt operation is used 
to substrtute the letters "re" in "compress" for the letter "a" in "compass." Note that the backtracking an-ow in Rgure 21 
spans one row but two columns, thereby signifying the multicharacter (merge) sul5Stitution. Hence the edrt distance is 
one, not two, in this case. By way of conparison, Rgure 20 illustrates the basic edrt distance algorrthm without utilizing 
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the two new multicharacter operations. Thus the cost (as depicted in Rgure 20) of editing "compass" into "compress" 
is two. 

The above-described procedure works well in most user-drawn annotation applications. The combined use of vec- 
tor quantizing and edit distance computation yield a system that is remarkably robust in its ability to find matching 

5 strings and substrings, even If they are not drawn precisely the same way by the user. Although the presently preferred 
embodiment has been illustrated here, a number of variations are possible without departing from the spirit of the inven- 
tion. For example, if a faster match is desired, the system may perform an initial lirst pass" match by simply finding all 
strokes that have a similar nun*er of data points. This may be done by storing the rujmber of data points as part of the 
feature data and then simply selecting or excluding those strokes that are not within a predetermined data point count. 

10 This type of first pass search can be performed quite quicWy, as simple numeric matching algorithms are all that are 
required. The first pass technique based on data point count would not, however, allow matching substrings to be 
extracted as the edit distance computation permits. Where higher matching accuracy is desired a more computationally 
costiy matching technique such as a Hidden Markov Model technique may be used as a final pass on the n-best hypoth- 
eses determined by the edit distance conrputation. Adding a highly accurate, but computationally costiy processing 

15 stage to the final output may be used in systems where it is necessary to discriminate between a targe number of highly 
similar strings. 

In summary, there is disclosed a system where the user 102 communicates through a digitizing writing surface 26 
with the audio/video control apparatus 20. An on-screen display 32, 34 is generated, providing the user 102 with a user 
environment in which a wide range of different tasks and functions can be performed. The digitizing writing surface 26 

20 can be incorporated into a hand-held remote control unit 24 and the audioA/ideo control apparatus 20 may likewise be 
incorporated into existing home entertainment or computer equipment. By tapping on tiie writing surface 26 a command 
bar 32 is presented on ttie screen, allowing the user 102 to select among various functions. Included in these functions 
is an on-screen programming feature 158, 148, allowing the user to select programs for viewing or recording by entry 
of user<Jrawn annotations or commands via tiie writing surface 26. 

25 The foregoing discussion discloses and describes exemplary embodiments of the present invention. One skilled in 
the art will readily recognize from such discussion and from tiie accompany drawings and claims, that various changes, 
nxxiif ications and variations can be made therein without departing from the spirit and scope of tiie invention as defined 
in the following claims. 

30 Claims 

1. An audio/video system having an enhanced video user environment, characterized by: 

an audio/video control apparatus (20) for selectively performing predetermined audio/video control functions 
35 (106) in accordance with a user's (102) selection, said control apparatus (20) including a port (62) for coupling 

to a video display apparatus (64; 36; 22) for displaying video material; 

a remote control apparatus (24) having a digitizing writing surface (26) for entry of hand-drawn instructions by 
a user (102), said renx)te control apparatus (24) communicating witti said audio/video control apparatus (20); 
a processor (72) communicating with at least one of said audio/video control apparatus (20) and said remote 
40 control apparatus (24) for controlling operation of said video display apparatus (64; 36; 22) in accordance witti 

said hand-drawn instructions. 

2. The system of claim 1 , characterized in tiiat said remote conti-ol apparatus (24) comprises a hand-heW push-button 
remote control structure (54-58) with said digitizing writing surface (26) incorporated into said structure (54-58). 

45 

3. The system of daim 1 or 2, characterized in that said remote control apparatus (24) communicates with said 
audio/video control apparatus (20) by infrared signals (30). 

4. The system of any of claims 1 - 3, characterized in that said remote control apparatus (24) communicates bidirec- 
50 tionally witti said audio/video control apparatus (20). 

5. The system of any of claims 1 - 4. characterized in that said rerrxjte control apparatus (24) includes a microphone 
for input of speech instructions. 

55 6. The system of any of daims 1 - 5. characterized in that said digitizing writing surface (26) is responsive to a hand- 
held stylus (28). 

7, The system of any of daims 1 - 6, characterized in that said digitizing writing surface (26) is responsive to the user's 
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fingertip. 

8. The system of any of daims 1 - 7, characterized in that said audio/video control apparatus (20) includes at least 
one control port for coupling to at least one connponent (36-52, 66) of audioMdeo equipment and wherein said 

5 audio/video control apparatus (20) Includes a control module for issuing (68; 70) control signals through said con- 
trol port to said component (36-52; 66) of audio/video equipment. 

9. The system of daim 8, characterized in that said component (36-52; 66) of audio/video equipment is a component 
selected from the group consisting of television (36; 22), video cassette recorder (VCR) (46), audio tape recorder 

10 (44), audio disc player (48), video disc player (48), audio amplifier (42). sun^ound sound processor (38. 40), video 
signal processor, camcorder (50), video telephone, cable television signal selector, satellite antenna controller, 
computer (52), CD-ROM player, photo CD player, video game player and information network access device. 

1 0. The system of any of claims 1 - 9. characterized in that said processor (72b) is disposed in said audio/video control 
15 apparatus (20). 

1 1 . The system of any of clainr^s 1 - 9, characterized in that said processor is attached to said audio/video control appa- 
ratus (20). 

20 12. The system of any of claims 1 - 9. characterized in that said processor (72a) is disposed in said remote control 
apparatus (24). 

13. The system of any of claims 1 - 9. characterized in that said processor (72) comprises a multiprocessor system 
(72a. 72b) having a first portion (72b) disposed in said audio/video control apparatus (20) and having a second por- 

25 tion (72a) disposed in said remote control (24). 

14. TTie system of any of daims 1-14, characterized in that said audio/video control apparatus (20) includes an inte- 
grated television tuner for tuning a user selected channel carrying program infomiation and providing a video signal 
representing said program information to said video display apparatus (64; 36; 22). 

30 

15. The system of any of claims 1 - 14. characterized in that said video display apparatus (64; 36; 22) is a television 
(36; 22) and wherein said audio/video control apparatus (20) outputs a video signal through said port, preferably 
an IMTSC or PAL or HDTV signal. 

35 16. The system of any of claims 1-15. characterized in that said audio/video control apparatus (20) is incorporated 
into a component of audio/video equipment. 

17. The system of claim 16. characterized in that said component of audio/video equipment is a component selected 
from the group consisting of television (36; 22), video cassette recorder (VCR) (46); audio tape recorder (44). audio 

40 disc player (48). video disc player (48). audio amplifier (42). sun-ound sound processor (38, 40), video signal proc- 
essor, canrrcorder (50), video telephone, cable television signal selector, satellite antenna controller, computer (52), 
CD-ROM player, photo CD player, video game player and information network access device. 

18. The system of any of daims 1-17, characterized in that said processor (72) includes a speech recognizer nxxiule. 

45 

19. The system of any of claims 1-18, characterized in that said processor (72) generates at least one meruj (32, 34) 
of user selectable system control options (136. 156. 160-168) and said audio/video control apparatus (20) issues 
a signal through said port (62) to display said menu (32, 34) on said video display apparatus (64; 36; 22) coupled 
to said port (62), wherein, in case said processor (72) is a multiprocessor system (72a. 72b), at least one processor 

50 of said multiprocessor system (72a, 72b) generates said at least one menu (32, 34). 

20. The system of any of daims 1-19. characterized in that said processor (72) is coupled to memory means (74. 76. 
86, 88) for storing user input. 

55 21 . The system of daim 20, characterized in that said user input comprises handwritten annotations drawn on said dig- 
itizing writing surface (26). 

22. The system of claim 21 , characterized by an on-demand video interface (158, 148) whereby said handwritten anno- 
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tations are used to recall a prerecorded entertainment program for presentation on said video display apparatus 
(64; 36: 22). 

23. The system of claim 21 or 22, characterized in that said handwritten annotations are translated into a known com- 
puter character set for subsequent processing. 

24. The system of any of claims 1 - 23, characterized in that 

said digitizing writing suface (26) is a digitizing writing display surface for entry of said hand-drawn instructions 
by said user and for displaying information to said user; and 

said processor (72) is a multiprocessor system (72a, 72b) having a first portion (72b) disposed in said 
audioArtdeo control apparatus (20) and having a second portion (72a) disposed in said remote control (24), 
said multiprocessor system (72a, 72b) communicating between said audio/video control apparatus (20) and 
said remote control apparatus (24) for controlling operation of said video display apparatus (64; 36; 22) in 
accordance with said harxl-drawn instructions. 



14 



EP0 838 945 A2 



20 



CD 



.34 



\ 




Figure 1 

15 



EP 0 838 945 A2 




16 



EP0 838 945 A2 




FP O 838 945 A2 




Figure 4 



EP0 838 945 A2 



72a 






^-76 


1 ^ 

Push 




1 n 

Push 




M 


ROM 




Button 
Display 




Button 
Keypad 




1 

C 



-74 



Processor 



IR Interface 



IR Interface 



Processor 



-80 



.78 



.90 



92. 



4- 



88 



Video 



72b 





Tablet 




Tablet 




Interface 





82 




20 



Video Out 




62 



RAM 



ROM 



Figure 5 

19 




114 



112. 



100 



110. 



108. 



i 



Hardware 



A 




^ User Interface j 




Applications ^ 







" Drivers 
and Libraries 





Microl<ernei 







Hardware 
Abstraction Layer 





106 



Electronic Program Guide 
Video Player 
Multi-User Games 



Input 
Video 

Network Management 



Small Real-Time 
Operating System 



< 



Timers, Tuning. Video, 
Graphics, Security, 
Peripheral Control 



104. 



Figure 6 



20 




21 



EP 0 838 m5 A2 




22 



EP 0 945 A2 




23 



EP0 838 945 A2 



c 

g: 

CO 



CO 
CO 



o 



> 5 3 <3 



♦ ♦ ♦ £ 



OOI <u 

Q J 5 S « 

3 o 2 « to 

> K CD 03 



m 






< 




to 


oo 


► 








o 



c 

Ql 

o 



10 

E 

CO 

CD 



u 
> 



^- — 



CM 



cu 
o 



24 



P 0 22S 94*5 A2 




25 



EP 0 B3b 94o m2 




26 



ZP 0 B38 A2 



i'®|@|g>|@l<^s 

«^ z Q u 

V □ S □ ® © 



□ 



E 
a 

o 

tn 




@ 

= G 


® 

tfl 

II 










2:00 pm 


World Series 


Friends" 
(cc) © 


Seinfeld 










1:30 pm 


X 

T3 

o 






h 








1 :00 pm 
















12:30 pm 








c 
c 

CO 






•J $ 

u 6 
|S5 



0 



0) 

c 

c 

X 

u 



c 
u 



U 01 

c c 
c c 



u 



o 



= i 
6 6 



C 



cn 
c 

Q. 
O 

x: 

CO 

c/) 

E 

to 



0) 

o 

Q. 



00 



CD 



CD 



27 



PP0838 945 A2 




28 



EP0 838945 A2 




29 



EP0 838 945A2 




30 



EP 0 A2 




31 



Incoming 
Ink Data 




200 



Figure 18 



T 

Break 
Into Strokes 



-|o 


— 1 


Ex 


tract 



202 



Feature Vectors . 



204 




206, 



Classify 'Compare 
Stroke Jypf ^ Feature Vector 



T 



'for Input Stroke 
to All Cluster 
Centroids 



Output String 
of Stroke Types 




216 





Centroid 


hg)i IV 


\ Type 1 Strokes 
J 'Vertical Lines" 




Centroid 




A Type 2 Strokes 
7 "Horizontal Lines" 


214-^ 


Centroid 


• 
• 
• 


] Type 3 Strokes 
J "circles" 







(27.2, 14. 0.933. ...) 


Ink 




Feature Vector 



210- 



One Pen Stroke 



218 



32 



EP 0 838 94:> a2 




33 



ZP C 235 945 ^2 




O 



O CN CO ^ lO 



O <N CO lO O 



o — CNco^in<Di^ 



o o E a o 
Aj9no 



CXJ 

cu 

CD 
Li- 



c 
o 

3 
o 
c 
c 

< 

(D 

o 

CO 



o 



— »— CM <N CO CO 



r- <N CN CO CO 



— CM CM CO CO CO 



r- r— CM CM CO CO CM 



r- CM CM CO CM 



^ r- CM CM CM CM 



^ CM r- CM CM 



CM CM 



r- r- O I— CM CO 



^ n- O «— CM CO 



/ 



/ 



/ 



/ 



r- O CM CO CO ^ 



O »— CM CM CO CO 



r 



CM CnI CO CO ^ 



»— r— CM CM CO CO CO 



r- CM CM CO CO 



f— CM . CM CO CO 



r— r- CM CM CO CO CO 



^ ^ CM CM CO CO 



r— ^ CM CM CO iO 



.— CM CO lO O 



O*— CMCO'^iOOr-- 



O 

CO 



o E a o to 
Ajano 



c 
o 

o 
cu 

X} 
0) 
-C 

o 
75 



34 



